On Negative Examples for Distantly-supervised Relation Extraction

A careful reader of our recent EMNLP paper (Surdeanu et al., 2012) will observe a somewhat untraditional notation: we use Pi to denote the known positive labels for an entity tuple i, and Ni to denote the set of known negative labels for the same tuple, that is, labels for which tuple i serves as a negative example (introduced in Section 4). This is uncommon: traditionally at training time, an entity tuple (e1, e2) that does not exist in the training DB is considered a negative example for all possible labels, which makes the maintenance of an explicit set Ni unnecessary. Most previous work on this topic, including our EMNLP 2012 paper, used this heuristic. However, in the context of the KBP slot filling task, where most infoboxes provided as training data are incomplete, this heuristic is not ideal. For example, let’s assume that we have an incomplete infobox for Rachmaninoff with a single slot (person:country of birth, Russia), and during training we see the tuple (Rachmaninoff, United States). What label should we assign to this tuple? According to the above heuristic, this tuple is a negative example for all the valid labels for PERSON entities, including person country of death. But this is wrong: in fact, Rachmaninoff died in the United States. We just did not have this information in the corresponding infobox. A better heuristics, which, to my knowledge, was discovered at the same time by (Sun et al., 2011) and (Surdeanu et al., 2011), is to consider the tuple a negative example only for the labels which exist in ei’s infobox with a different label. More formally, using our notation, Ni for the ith tuple (e1, e2) is defined as: {rj | rj(e1, ek) ∈ D, ek 6= e2, rj / ∈ Pi}. That is, for the above example, (Rachmaninoff, United States) should be considered as negative example only for person:country of birth because this is the only thing we know with certainty about Rachmaninoff in our training dataset. Modeling this heuristic using local, one-vs-rest classifiers is trivial, as illustrated by both (Sun et al., 2011) and (Surdeanu et al., 2011): you just create different negative example sets for each label, according to the data available in the infoboxes. However, when dealing with a joint model, implementing this is not exactly trivial.1 To pat ourselves on the back, we designed our EMNLP algorithm from the very beginning under the assumption that one needs to maintainNi explicitly, although we left the empirical analysis of the heuristic discussed here as future work. So our algorithm works as is with this new heuristic (you just create Ni differently).